Papers with cross-task generalization
Systematic Analysis for Pretrained Language Model Priming for Parameter-Efficient Fine-tuning (2024.naacl-srw)
Copied to clipboard
| Challenge: | Parameter-efficient (PE) methods for adapting pre-trained language models to downstream tasks are still lacking in many cases. |
| Approach: | They propose a general PE priming framework to enhance few-shot adaptation and generalization ability of PE methods. |
| Outcome: | The proposed framework reveals that the best priming strategy facilitates adaptation to target tasks. |
Cross-Task Generalization Abilities of Large Language Models (2024.naacl-srw)
Copied to clipboard
| Challenge: | a thesis proposal advocates for the crucial role of cross-task generalization in NLP systems. |
| Approach: | They propose to benchmark cross-task generalization abilities with diverse NLP tasks . they also propose to develop model architectures for improving cross- task generalization . |
| Outcome: | This paper compares cross-task generalization abilities with diverse NLP tasks . it also analyzes and predicts the generalization landscape of current state-of-the-art large language models . |
InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Instruction tuning is emerging in NLP, but has not been explored for dialogue-related tasks. |
| Approach: | They propose an instruction tuning framework for dialogue that leverages natural language instructions with language models to induce zero-shot generalization on unseen tasks. |
| Outcome: | The proposed framework enables good zero-shot performance on unseen datasets and tasks such as dialogue evaluation and intent detection. |
POSITION BIAS MITIGATES POSITION BIAS: Mitigate Position Bias Through Inter-Position Knowledge Distillation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Positional bias (PB) manifests as non-uniform sensitivity across contextual locations . previous studies have addressed PB by modifying the underlying architectures or employing extensive contextual awareness training. |
| Approach: | They propose a position-to-position knowledge distillation framework that leverages position-induced disparities to counteract PB. |
| Outcome: | The proposed framework reduces positional bias and improves performance on retrieval and reasoning tasks. |
Self-Specialization: Uncovering Latent Expertise within Large Language Models (2024.findings-acl)
Copied to clipboard
Junmo Kang, Hongyin Luo, Yada Zhu, Jacob Hansen, James Glass, David Cox, Alan Ritter, Rogerio Feris, Leonid Karlinsky
| Challenge: | Recent studies have demonstrated the effectiveness of self-alignment in which a large language model is aligned to follow general instructions using instructional data generated from the model itself. |
| Approach: | They propose to use human-written seeds to align large language models to follow general instructions to achieve cross-task generalization. |
| Outcome: | The proposed model outperforms base models and models that are generally instruction-tuned or have been adapted to the target domain by a large margin. |
Beyond Full Fine-tuning: Harnessing the Power of LoRA for Multi-Task Instruction Tuning (2024.lrec-main)
Copied to clipboard
Chunlei Xin, Yaojie Lu, Hongyu Lin, Shuheng Zhou, Huijia Zhu, Weiqiang Wang, Zhongyi Liu, Xianpei Han, Le Sun
| Challenge: | Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning algorithm for large-scale language models. |
| Approach: | They conduct a systematic study of Low-Rank Adaptation (LoRA) on diverse tasks and rich resources with different learning capacities. |
| Outcome: | The proposed algorithm can achieve remarkable performance in high-resource and multi-task scenarios, even comparable to full fine-tuning. |
Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2022.acl-long)
Copied to clipboard
| Challenge: | Despite the success of supervised learning, models often struggle with generalization across tasks. |
| Approach: | They propose to use crowdsourcing instructions to build a model that learns a new task by understanding the human-readable instructions that define it. |
| Outcome: | The proposed model can learn from seen tasks and generalize to unseen tasks given its natural crowdsourcing instructions. |
Instance-Level Dynamic LoRAs Composition for Cross-Task Generalization (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models perform well on tasks that have undergone fine-tuning of instructions, but performance on completely unseen tasks is often less than ideal. |
| Approach: | They propose a task-level LoRAs combination which learns the LoRA modules combination weights based on a small number of samples to form the task model. |
| Outcome: | The proposed method outperforms the typical method, LoraHub, on 16 out of 27 tasks. |
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022.emnlp-main)
Copied to clipboard
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, Xudong Shen
| Challenge: | a benchmark of 1,616 diverse NLP tasks and their expert-written instructions is used to test generalization of models to unseen tasks . a recent study shows that instruction-following models outperform instruction-based models by over 9% . |
| Approach: | They build a benchmark of 1,616 diverse NLP tasks and their expert-written instructions. |
| Outcome: | The proposed model outperforms existing instruction-following models by over 9% on the benchmark despite being smaller. |
Enabling Natural Zero-Shot Prompting on Encoder Models via Statement-Tuning (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit remarkable capabilities in zero-shot and few-shot settings, but they struggle with extending to few- shot and zero- shot settings due to their architectural design. |
| Approach: | They propose a technique that models discriminative tasks as a set of finite statements and trains an encoder model to discriminate between the potential statements to determine the label. |
| Outcome: | The proposed method achieves competitive performance compared to state-of-the-art LLMs with significantly fewer parameters. |
Learning to Initialize: Can Meta Learning Improve Cross-task Generalization in Prompt Tuning? (2023.acl-long)
Copied to clipboard
| Challenge: | Prompt tuning (PT) based on frozen pre-trained language models has shown remarkable performance in few-shot learning . however, it relies heavily on good initialization of the prompt embeddings. |
| Approach: | They propose to use meta prompt tuning to improve cross-task generalization by learning to initialize prompt embeddings from other relevant tasks. |
| Outcome: | The proposed method outperforms PT on classification tasks, but not multi-task learning. |
HyperLoRA: Efficient Cross-task Generalization via Constrained Low-Rank Adapters Generation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to adapt pre-trained language models (PLMs) to emerging tasks are costly and inefficient. |
| Approach: | They propose a meta-network that generates task-specific weights without any optimization. |
| Outcome: | The proposed approach has flexible generalization ability and superior performance over hypenetworks. |
Table-R1: Inference-Time Scaling for Table Reasoning Tasks (2025.emnlp-main)
Copied to clipboard
| Challenge: | In this study, we explore inference-time scaling on table reasoning tasks. |
| Approach: | They propose a large-scale dataset of reasoning traces and a reinforcement learning with verifiable rewards approach to enable inference-time scaling on table reasoning tasks. |
| Outcome: | The proposed model matches or exceeds GPT-4.1 and DeepSeek-R1 models on diverse table reasoning tasks. |